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TITLE I PXD TtSB IKSTITUTICKALIZIKG 0? EVfOmridS 
IS P2i UR2AK SOKOL ^fSTEH 
/ » 



The focas opthis paper is to describe the intact on evaluaftioa 
* activities ia a lar^ urbsa schJ^l system pr&cic«d by the e^-aJuatian isaij- 
date of the' nleiDen^liy a:i3 Secosrdaiy Bdacation Act ox I56S for Title T 



* pro^grsms Froo the vantage point of a decade of sudj eval^satioa efforts 
for ESL^ Title I, it is clear t?iar aa iJSiauticipated result ia the scfcool 
district has been the legitiiuzintg of a polic>' for e%-aluatioa. During 
this decade of evaluation experience, fomialatioa of evaluatica policies ^ 
and injrleffientatioa of evaluation activities occurred^^espi^^^^e cxsistraints 
produced by the CDnplexities of the district bureaucracy. 7^at»thc izEpetiiS 
for deternlning progran efforts could be sustained, although viewed some 
district authority figures and practitioners as a necessary evil for quali- 
fying for funds > represents a r^re turn of events. It runs counter to the 
typical fate of evaluation in bureaucratic systems. 

School district decision icakers traditionally have not accorded 
evaluation findings an iii5>ortant role, if indeed any portion of a role, in 
the formilation of policy or prograinaing. Little consideration has been 
given to the data jieed^ to sustain prograia. worth* Few -denands have been 
presented for. objective evidence of program effects, Researdiers .of 
innovation processes have contended that the administrator's tendency in 
the matter of implementing educational change laore often than not has b^en 
to •^fall back on bromides, rules of thumb and conjceptuali^tions without 
a strong ei:5)irical base" (JGies, 1974)* , . ' 

/ There have been, however, forsddable problems in maintaining 
evaluation activities ♦ Caro (1971) asserts that freexy available facts 



jfiight redace the security of the boresacrats who waald view e^'aluative 
data abxit prograns as a threat. The Iok prestige accorded to research 
in action settings, the icylereatatica problens pf ^research in action 
settings aad disagreenent aboat cse of results feaye also ccntriimted to 
the prrfjleffi of naintaining evaluative researdl activities. FrosB an 
extensive rerievf of the literatare cn isstses in evaitjaticn of social 



progT23is, Caxo fi97ij identifi^ iiajrr prcM^s^j^gt-ferrf placed ad- 
xdnistrators sad evaluators at crt^ss purposes-i — 5^^e include ii^effective 
conmunicatioa and collabaratioa between administrator and evaluator, 
vague definitions of roles, conflict between staff and nanagent^ loyalties 
burdensome data collection procedures, issiies of adudnistrator progran 
ccinmitBsnt and ek'aluator progran nejJtrality, loose dissendnaticn of 
results and budget constraints. . : 

In 1965, the candate for evaloaticsj of ESEA Title- 1 prograiES was 
ur^nrecedented. Since tha^ tiioe, disensions of the failures ai^ successes 

r 

-groduced by the jsandate have provided a major thecre in the literature 
related to evaluaticm. Cohen (1970) in his analysis of the.ESEA'Title I 
evaluation provision noted that it lacked any ^enabling isechanisia" and 
that the responsibility for carrying it out was specifically delegated 
to state and local educaticm authorities who operated the progxaos*^ Ihis 
Varti of events was, in his opinion, "equivalent €o abandoning Euch hope, 
of useful progran evaluation*" In addition, Cohen (1970) erphasize^ the 
political diaensi<m of evaluation. The politics were cnXxcsX because 
the use of in&xration whidi was at least potentially relevant to 
dedsion-raki ng was involved* Such efforts would result in allocation 
o£ resources- -xxmey, position, authority involved and the li3ce» To the 
extent that this infoTceation was a basis or excuse ^laxij^ng po^ex 



relati^iships vithia the iastitiztioa, the evaluation, in Coben^s opiidxn, * 

becsme a political activity. 

In a. thorough case study of ESEA Title T^^^ijation injleifien- 

tat ion, HcLaiighlin (1975) described the efforts as "a isixture of refora 

coanxerrefom, dejnaad and cDnproiEise^ *fhicii vpere generally dissipated 

by the "constraints of a polfcy s^^tea and b^Jiayjor of bureaucracies 

^McLaughlin {1975} identified najor ol^tacles to the effort: 

♦ Tne history of Title I evalizaticn also suggests a nuniber 
of iiiplicatioas about the ccndact and use of evaluation 
in a jHUlti- level go^*en^ent structures^ These lessens raise 
both nethodological and functional questions about the 
vdsdoia of continuing pus^suit of scientific rationality, 
especially in the instance ox broad- aim social actic^ 
pr o gi anis , such, as Title I, vhich x - epre sgnt an instituticxial- 
ized federal investment. The Title J experience has- shown 
how resistant the educational policy systen is to assessment 
of achievenents az>d accoii5)lishinents, and also that a ntinber 
of obstacles to this ccmfinsatory style of r^jorting.are 
irfierent in tJie systen itself .Specifically, data on the 
relative effectiveness of teaching strategies or allocation 
of xesources vil4 be difficult to gather not coly because 
of the unsystenatic and decentralized data ^collection 
practices existing at the local. level, but also because 
local prograins have little iirjterest in these ^data and aare 
disinclined to collect theia or furnish them. 

In McLaughlin's view, the final crux of the parobl^ in the^ 
Title I experience vfas that ''anyone who looks to evaltiation to take the 
politics out of decisionmaking is boimd to be disappointed*" * McLaughlin 
(197S> contended; "Evaluation efforts bascKl on ejcpectations for r^forn 
by Deans of a social report, or better infonsation on progran accocplish- 
isent^ certainly would find 'justification in theory^ Eut in practice^ 
they cay turn out to be little more than esgrty ritual." 

Observation of^puMic school affairs in the ten yeajss following 
passage of ESEA has prompted Halperin (197S) to identify various educa- 
tionai'"successes for Public Law 89-10/ He attributed the beginning of the 



edacational sccoaatability roveiDeat to ESEA Title I evalastioa provisions. 
, •*Ia IBSiS,^ he noted, ♦'any federal prescription for testijig, for ceasizriag 
ed:acat4onal oatcones, for requiriag aaything other than basic fiscal 
honesty, was certain to provide rrTTTTrnl tion for £SEA*s foes.** Althos^ 
£S£A. eyalim^ioas got off to a slow start with \fide .progran ^^ariaticns 
a ron g States, no caifona reportiixg procedures, few cocparable practices 
xo as*sess,^-cjeaxcut xrazidates for evaiuaticai ijf e&caticaal progrscs have 

' ^^^^^ n 

emer^* The cost rtxx^nx extezisioa of ESEA (^.L* in Ralperin's 

(pinion, showed that the Congress has embraced the cozx^t of ei^aluation 
with its stipulation for develc^nient azid publishing of stafiaards for 
detemiaatioa of effectiveness. Tnis xequirenent, he contended, wduld 
**havfe been politically unthinhsLble a decade ago.** 

Although the literature generally has featured the failures of 
natic^l' evaluation effo;t^ of Title I, rarely has it included accounts of 
local Ti^e I eralpatitm effort^. Yet the candate for Title I evaluation 
engendered" a range of responses froa local educational agencies* O^ehster 
1972) ♦ Sonie SOO school districts* are currently operating research/ evaluation 
units* ?^ Certainly sojae Jiomentua for this state of affairs can be attributed 
to the need to produce Title I evaliiatioas. 

Kebster (1972) observed that the presstare for evk^uation iin the 
piAlic schools/ in the absence of an earnest interest objective infbrzca- 
tion, led to inany research and evaluatio departiaents structure along 
^project lines. In his^*<^i^on, such structuring had tneded to inhibit the 
establishiaent of a systeriatic data base and to discourage the adoption of a 
unified strategy, for evaluation. In Cleveland, the responsibility for ESEA 



^ Personal cassamication, Tn.,lzxTf W. Barber, Vice President, SchooJ 
Evaluation and Frogrzn DevelGjoent (Divisicq /isexican Educational^ 
Research Association.) . ; 



Title I rraluatica fell to the Sareau of Educational pj&search* Siace the 
early t:wenti^s, the Bureau was a service division uithin the Cleveland systeiL 
Traditiraally, the Bareau provided^ iafoncation retrieval related to selected 
areas of ft^idnistrati« research as veil as city-vide testing services. 
In this latter instance, the Sure^ staf^ had constrocted auiaerous types of . 
tests and producai local nonss for these instruments to gauge ijipact of 

4 

varioos Cleveland curricola* A regular testing prograa for eleneataiy stAools 
had been inpleanent^ amrually six>ce the 1923-24 sciiool year* Flessixtg, 1974) 
notes that under the direction of the then Divisicm of Reference aini Research, 
the original plan irr»*olvSi adiEinistratian of initial and final tests, in order 
to facilitate study of the semester's growth in achievezaeat ia the five 
acadeudc subjects in which, the tests were given. Such a testing aarran^ement 
correlated vith the semi-aimiial pronotion ^lan of the Cleveland systea in 
effect 'at that tiaae. 

To meet the d^sands of evaluatioir services for ESEA Title I, the 
first task for the Sureau became the recruitment of a staff vith coii5>etencies 
in evaluation. Develq>ment of the prc^sals--^jplications, narrative^, 
budgets and the like, emerged as another priority for the Bureau's ser^ces, 
Froo the onset, the » strategy .for pro^aa development %ca$ the requirement 
that the design of the program be scheduled concurrently with the design of 
evaluation procedures. A major advantage of this proceduare was readily ap- 
parent^ a cohesive approach for programaihg and eval'txation was developed-- 
producing a more efficient use of the resources that became 'available. 

Within two years of the mandate, the Bureau became the Division 
of Research and Deveiopment reflecti^ the new jgriorities for its services 
vithiA the sdiool system. 

As a bacJ;grouiSd for consideration of the iaf oma tion ^ather^ 
^bCRit the patt^ins of evaluative services provided in the <Hstxict, it is 



veil to consider the «track recortT of Title I projects vithin the district. 
A total of 32 sepB±-$tJt projects have beeir funded trader Title I ?ince 
Of the 11 projects, currently in opdratioa, three or slightly rore than 
nine perceot^originate4 duilng the first year of t^ration. Toe district s 
style for installation of ixev ^Jrojects has generally been a oae-at-a-tirie 
plan, except for 1967-156S when four of the 11 currently. c^rating projects 
wpr^ rrrigi R ated- the su3rvivaa rate for' the projec ts 1^ bera snnniarized 



in Figure 1* 

As can be seen frois Figure 1, the suridval rate of Title I projects 
has been rather substantial in the district • Fourteen or alnost 44 percent 
of the projects have survived through being t rgns£erre d intact to other 
funding v*ere th^ have contiiwed to operate. The reasons for sudi trans^ 
ferral have been xsany. In nost instances,^ transfers vere laade because 
other specialized funding sources became available vhicfa were directly in- 
tended to svci^poTt tkese efforts. Sonfetines, the target jxjpulations in these 
projects were iiiore generalized than categorical as was requijfed uoier Title I 
guidelines- Other reasons for transferral involved the type of services 
being provided under Title I. For exanple, staff develc^nient efforts were 
given little priority in the state regulations. The directives that Title I 
services should be targeted on the early years of the sdiool e3q>erience and 
that the projects mxst involve direct service to children rathei: tiian efforts 
to train teachere case throu^ loud and clear-p 

Knen taking into account the transfer policy of the projects, the 
survival rate of Title 1 project^ in the digtrict has been hi^. It £s 
■ to say, hcwever, that the a<iainistrative experience'gained in operating the 
Title I projects was a factor in such de^fisions.' In -addition, izifpfration 
frop the evaliations docoaented that the activities vere serving childfen 
a»d teachers in perfectly legitimate' vzys a^thon^ in conflict vith Title I 



FIGURE . 1 
SURVIVAL RATE 
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gliidelilies. Generally, the evarluation% .provided data not only aboiit what 
the 'staff said it was doing, but what the other audiences said they were 
Joing. Consideration of such a variety of perspjsctives helped to clarify 
|hat was happening and what should have been happening. It -has been a 
Local fetish from the on^tart of evaluation activities to atteiiqpt to relate 
piq)il outcomes to v^ether or not the treatment was delivered. Without this 
base, pronouncebents about client change vere meaningless. From such a 
cauldron of adoinistrative experience and evaluation findings, a modus' 
tl operandi has emerged. To the credit of the district > the art of project « 
development and ic^^lementation has come a long way since 1965. 

That there are some ^'better ways than othets" has been demonsti^ated 
in the matter of implementing projects. Fo'r example, projects of some scope 
require shakedown periods. For a major project, this time can mean about 
three school years to implement and refine operations. During, this shake- 
down period, communication of project intent to school and community is 

4 

critical. Credibility is an issue here. Pupil selection processes, schedul- 
ing of services, and, of course^ selection of the staff require careful 
attention during the installation stages. The staff must be committed and 
they will need time to learn th^ir roles. These phenomena in the educa- 
tional enterprise have been identified time and again in the various 
evaluations provided for Title I and other projects. Unfortuna^tely, these ^ 
^ issues seem to be rarely considered by legislators and bureaucrats. They 
are usually most disillusionded when instant improvements are not forth- 
coming and do not ?eem willing to deal with the realities of project 
implementation. 

Another consideration in the background of the impact of the 
Title I mandate for evaluation in the school district is the experience 
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with the evaliurtion liaison teacher plan. This effort was an attempt to 

upgrade the competencies of classroom teachers in pfogram evaluation at * 

the local. school level. It was undertaken 'by the Division of Research and" 

Deveiopement in 1971. This plan provided for assignment of an^ evaluation 

t ' 
teacher at each school to serve as a resource for the school staff in 

assessment of its programs. The model wa^ first implemented in connection 

with the 'summer school programming offered in Cleveland's Title I scho61s» 

Each school staff had responsibility for undertaking planning and develop-- 

ing a summer program. As a resul.t, evaluation was also undertaken at the 

local school for the suismer school program". Consultants from the Division 

of Research and Development provided assistance to the teachers via the 

phone or* on sxte.' The teachers were responsible for supportive services 

to the staff in the areas of development of instructional objectives', 

data collection, data analysis techniques and reporting procedures. The 

Div|?sion staff developed four training sessions for the teachers and, any 

principals who wished to attend. An evaluation packet of practice and 

source materials was provided for each school. 

The evaluation liaison teacher model proved workable-r-indicating 

that local school personnel can evaluate, that Jargerscale training pro- 

grams in evaluation are "feasible and that support through central staff 

consultation produced more effective programming (Fleming, 1972). This 

experience served to involve every Title I school iii evaluation efforts 

and to demonstrate the p;racticality of such . efforts • 

To assess the impact of the ESEA Title I mandates upon the evalua- 
* ' • . ** 

tion activities of the Division, twcJ approadhes were employed for this 

paper. First, a case study vas designed to describe the evaluation 

activities of the Division observed at. two points of time. Although 



such a study has an ex post facto character, it attempted to document the 
requests for evaluation service and origin of such requests between two 
points in time. In particular the request rates for the two periods, the 
types of evaluative questions and the types of processes to be implemented 
were described. The case study was an atteii5)t to observe regular behavidr 
in the ordinary setting of the ETivision^s services. ^ The second approach * 
was. a survey of special project .managers and curriculum siqjervisors. 

Two geheral purposes provided the focus for the cas e^ ^ gtu d y of > 



the pattem^^of evaluation requests inade of the Division of Research and 
Development serving the Cleveland Public Schools. These purposes included 
(1) describing the nature of the requests for evaluation and (2) comparing 

the incidence of change occurring in the requests between two coii5)arison 

«' > • 

points' of time. * , ' * ' 

To obtaiii 'data to compare the nature of the evaluation req[uests, 
a simple form with 12 "Categories of seiyices was deviseS to classify tlfie 
requests. The classifications w^e derived from the published functions^ 
for the evaluation section of the Division. A record of the requests for 
the. various services was obtained ,from a monthly planner log maintainied / 
in the office of the Di^sion Director. The rec(Uests were redord^d with 



the intention of establishing two samples so that the overall pattern of 
the requests and request rates could be determined an<f compared. /Only 
initial or new requests jyere recorded in an §ffort to'idfentify developing^ 



trends and to distinguish such /activities from maintenance effoj^p^T^^^^ 
projects funded through Federal and State sources receive th6 evaluative 
services listed in the 'checklist as part of the on-going .operation of the 
.Division. The two points pf time selected were the 1971-^2 and 1974-75 
school years. These years were selected because 'they represented stages 
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of the Division's developcent . In the first instance, the Division re- 
ported to ti>e Deputy Superintend«Jt of the district. In the latter case, 
the Division had been incorporated into the D^xaxtnent 6f Special Projects 
and Dxttirmi rig Educatio:?. 

The staff i^eq^esting evaluative services vere classifi^l into 
those in the CurriQulun bxA Instruction D^artDent and those in the various 
con5>easatoTy . special projects. It should be not^ that these central 
staff personnel^ were c^rating spider the general fend in the first instance 
and federal and/or state sources in the letter in^^ce. 

Figure 2 5XJ3i23:am ^es the percentages^ of reqtiests for the vaxious 

types of evaluative services froa the central staff working in curriculua 

and instruction and those working in the varlotis projects. For the first 

cosparison period- -63 new requests were received froa C § I Staff. The 

reqtiests for the second period nuira>ered 78. ^)ecial project staff requests 

totalled 43 and 45 respectively. p 
> 

Comparison of the percentages of these; new requests for evaluatic^i 
indicates the •following: 

1. The types of requests froa C 5 I personnel reflect less 
V • change than the types observed froa the-SP staff. 

2. Since the first observation tine, C 5 I reqtiests" for 
consultation services have aliaost doubled; Other in- 

- creases occurred in the requests for training sessions 
and for develojHaent and/or ref ineaent" of objectives. 

3. New I requests for dc^eli^xnent of assessDent 
designs were'contimi&ir at a comparable rate, 
repTesentin^^_^4^ttiy core than one ^ut of five of 
the tota|^?3a^ests at both conj^arison points* 

, 4. Tlfe^^eatest decrease in C § I requests occurred in 
i^t^^ area of critiquing data collection requests* 



>^f ^^ C^iis last trend ray be attributable Vff a policy for conducting ^ 
; * . . 

toaperatLvB areseJarch studies itbich was .adopted by the sch<>ol systea in 1972. 

/ 

JJiest jfequests ai3? now dianneled directly to the Division*) 
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S* SP staff requests for presratstiozis of fisdiiigs.to . 
cu i ii i irnlt y groqps reflected the largest ii^crease 
between the two conjjarison poiixts. 

6. Presentation of fiBdings to counsxiity represented 
the seccad largest increase in SP* staff recpsests. 



7. Largest decrease ^^>eared in S? staff requests for 




exceeded all other categories in the second period, ^ 
Tt:B changes in patterns of requests froa tiie SP staS offer an 
interesting contrast to those observe for C € I personnel • Evaluation 
activities provided .by the Division generally &ve cnerged as a result 
of the requirenents of the funding source. The shifts 'in the nature of 
the reqiiests, therefore, appear to be related to the arandated procedures 
of the funding source* For e xainp le^ laore fomalired disseidnation of 
fin d i n gs' about , the projects is hcm required for the comsisnity* Aithongb 
cany projects have operated for a mcnber of years, project gui&lines 
have increasingly undated more specificity in prog r aa objectives which 
is reflected in a continuing level of activities in this* area in the 
second coEj^arison x>eriod. An increase also occurred in the aarea of analyjt- 
in^ data and ^;p lying data processing services. This derond sppax^tly^ 



is related td the increase av3i lability, of such services because of the 
^ school district's expansion of jts OTignxter services, 

The steady rate of reqjaests froa ^e Special project Staff in- 
dicates the cratinuing rer^le of activities in the projects in question- ^ 
The continuing and increasing pace of decand for evalt^atiCT services. fro» 
the C ^ I staff reflects a rippje effect froa the school ^^iStxiJcVs efforts 
to provide for the evaluation of Title' I 3si ofSber federal laSal ^taze 
fended i«roj^cts in an appropri ate Maimer* ' ^ ^ ' ^ 

Three purposes gave 4irecticb to ^e snanr4^ of Curriculum and 



TV'- 



Instoactka staff asti Sp^i^al Project naaagers. Thes« were (1) to identify 
the general level of satisfaction with evaluatica seivi^ces being provided 
by the Divisica of Research and De»*elc^>iaeat, (2) to describe the nature of 
the evaluation services khidi C 6 I staff reported receiving asd (5) to 
detemdne the changes in such serrices wbicih C £ I staff and project icanagers 
iti^t recomnend. . 

Sunreys were distributed tsirou^ the school icail' system to 2S 
C 6 I central staff and 4^8 SP nanagers. A resprase rate of 57 and 60 per 
cent respectively vras aoiieved. *the sur»*ey tonus vith percentages of re- 
sponse for the varioiis categories be found in- Appendix A, 

Alrost rxine out of ten C S I staff responding^ indicated that 

they had request^ assistance frcan the Divisica of Research and Develop- 

rent in 11 areas during the past three school j-ears^ ^ecificaHy, the 

following responses reflect the pattern of services provided. Over 80 

percent reported assistance in vriting project prograB proposals viii^e 

less than 20 per cent reported help in designing institxmentaticn and coa- 

piling infoncatic^. About forty-four percent rq>orted help in conducting^ 

a progran evaluation and aiiaost four out of ten J? 5 I respondents indicated 

assistance in writing behavioral objectives, conducting staff deve^c^ent 

prqgraits and researching particular 'tc^ics or questions . - * 

- . A highJ-eyfiJj^f satisfaiction was r^>orted with about nine out 
* ' ' O 

of ten ^rv»4/-^t^g ^ ^&ting of at least "well satisfied," Jle^)CTdents 

consented cn the pradieal, efficient service provided by responsive 

persons. Q^y one r^pondent indicated that ine;5>erienced personnel had 

been assign^' without sufStient training and guidance ftoa- the Division, 

liore half of thje respondents indicated that they vould definitely 

x^x^:jtst ^ rogra a evaluation service again frcfia ^d^e Division* 



Ihrec cnrt of four C 4 I staff reported baring received copies of 
rarioas special project eval oat ions asd ejgyressed positive reactioas to 
th^e r ep o rt s, tore than tyfo out of three of the C 6 I staff ii3dicated 
that they san these evaluation reports as rodels for evaluation of prograi^ 
in their c«rri culms area* Analysis of the coatent of their conneats in- 
dicates that ^ley vie^t the parpose of evaluation as providing information 
about whether or not prograns are ineeting their objectives so that de- 
cisions coa.W be nsde to in^ r ove the prograit* In addition, they appeared 
to view e'i'aluation as ideati^ing strengths aad ^^eainesses of prograns so 
that changes cc-ald be effected in the instruction services* for pig^ils. 

Three out of the seven recomn^daticEiS received fron C$1 sraff 
centered on the topic of personnel. Two called for expansion of the staff 
and the third, for assigning one person on a continuing basis so as to 
provide for continuity of philosc^hy and prograB airareaess. One respondent 
noted a concern that testing instruiuents should reflect the program cV 
jectives being evaluated. ^ 

The surv'ey data indicate that the C § I staff are Jsiowlesdgeable 
about the evaluation services available and .Express a high level of satis- 
faction with the evaluation services that have been provided. Iheir view 
^f the purposes of evaluation follows the general arationale p roml ^ted for 
c^nrpensatory progran evaluation. At this point in tiiae th^ see the possi- j 
trility of ^^3g>ortant information being generated for decision-Eaking. iSiis 
attitude apjpears to be substantiated *as well by the increasing level of 
special request, noted in the Division log, ♦ 

Definite contrasts in attitude ccald be predicted beti^n the 

z 

C 5 I staff and SP staffs In the latter case, -a rore negative vXem coald 
be anticipated because evaluation for the projects is jsandated try fending 



sources* Viewed zs a ^necessary e^i 1** and a|^rt>Dess for which the staff 
vrmild have no choice, rregativis^B ^ml6 "be the case more often than not. 
Contrary to this e3g>ectatioa , aliaost three o-Jt of fopur S? staff reported 
being at least ^Snell satisfied*' vith'the services of the Division ia 

- evaliiating their projects. Many nore concerns t>ian those e^^^ressed by 
The C 5 I staff were re^rted by SP personnel, however. 

Apparently experience with e^'aluaticn contribiJtes to greater 
support for the activity. About one out of four S? staff reloaded that 
their initial level of satisfaction inpro^'ed c^er the years as a resxxlt 
of becocang -Bore fairiliar iiith evaluation procedures, experiencing the use 

\of evaluation feedback, seeing an i j i p ro v ecent in the quality of the re- 
^►qrts and fmding a uore coqperative working relationship with a chapge in 
Valuators* More than one third responding saw the project staff as also 
growing ©ore favorable over the more recent years toward the evaluative 
protess. Factors contributing to the resistance or negative reactions of 
the ^ro^ect staff to evalixaticn were identified as "too icauy data collection 
procedures, inherent suspicion of evaluation^ poor connnunication about vhat 
was expected, tight schedule for data collection, lateness of availability 
of results asid the project staff* s difficulty in distinguishing between" , 
project evaluation and individual evaltiatJon." *^ 
Mhost 97- per cent of the ^ staff r^rted usin^ evaluation ^ 
findings in discussing the project in staff conferences and ©eetings, in ^ 
addition, findings were also disojssed in neetings-of t^ Parents' Advisory 
Councils, Parent Teacher Association^ district adaixilstrztoTS, state ax^ ^ 
federal officials and loc^l, state and national iseetings and daring parental 
visitations, toly ose xespondertt indicated not using the findings in any of 
these situations. 



Satis factdc?a vitfa sazae 17 aspects of the projects described in 
tfee project evaluations shoveed a range fron a lew of 14 per cent for tiie 
presentation of conssznity c^inion in the evaluaticq r^>orts to a hi^ 
of 79 per CCTt for presentation of the evaluation pi^ aiid data analysis • 
Other areas of ^^parent satisfaction included presentaticm of instroc- 
ti-onal procedures, psjxil selecticm and staff respcnsibilities. Relatively 
.Jess satisfaction vas indicated for presentation of cast effectiveness 
data and affective effects in the evaluations • 

Overall reaction to the evaluations provided for their projects 
say about a tvo out of three ratio of respondents selecting responses 
indicated as ^•enthusiastic^ usefej^, effective^ prc^^iag^ relevant, clear, 
interesting and positive" on*the rating scale. 

Analjrsis' of ccinmests of staff related to^jroviding infornatien 
for decision izaking about their projects show that tiiaeliness of the in- 
formation is viewed as determning its utility. Ifost view the data as a 
basis for adjusting plans and J'eorganizing op^xations* One respondent 
distinguished between decision rating for project operations and decision 
Esiing concerning the evaluatiiKi. 

The prirary purpose of evalua:^on of the projects vas vie^jed as 

identification of strengths and ireaSs^^es to determne whether *or not 

' ~~ « 

the project is jaeeting its objectives. Geceraliy^ reconoendations to the 

* *^ - . 

Bivision about* evaluation ^qperatior^ involve assigmarat of evaluation 
personnel who are re^>onsive and interact wiDth the jgrpject sta^, pro- 



viding «qpanded feedback in enou^ tiae to tie such infbnation &r 
i^yroveaent, Is^toved presentation of data, sij^l>fication of data 
collection ax^d irproved cdanunication about^e^eetaticns and xesj^sll^ities^ 
^ppa2«itly, tb6 assigning of perscGS as evalu^toocs i&o. ^e^lB 



to establish ^?propriate rspport aiA conmasxicatica is of critical iinpcfrtsnce 

in the design and carrying out of an evaluation. Particularly revealing 

is the following coioent froa a project nai^ger: ^ 

It is no secret that Mrs. X has been of inneasixrable 
st5?port to us, since the beginning, by: 

. interpreting project responsibilities to Title / 
. accocmodating, whenever passible, the special needs 

of the project 
. being available and Killing to attend laeetings of the 

entire staff viiene>^r invited 
. shoving enthusiasm for our wori and niaking every 

effort to present it in a positive naimer 
. being available for consultation at work, as well 

as at hose 

. arranging her work so that we know deadlines and 
data required far in adi-ance* 

Ke are very fortunate in having someone of'IferJ^sX^s 
backgroung and e^erience in the field of reading as 
✓ our evaluator. 

la their perceptive analysis of Title I failures and successes,. 

Ifeigbes and Hughes (1972) described qualities of the district's Title I 

progran. They identified its key features as centralized planning, 

deficit prevention, continuity of service, focus on achieving of grade 

level academic perforanance in relation to age and beaming of the entire 

' program at disadvantaged children from hard core poverty* Although 

quoting extensively from evaluation data to show that iicprovejBent in 

pxjpil behavior can be achieved if the prescribed treafeents are delivered, 

they take no note of *the fact that a concerted and fomalized -evaluation 

process h^ provided their infor3sati<m. ^ \ ' 

It- is evident from the incidence of reqaests for eyaluati^ 

services frtxa C § I staff and the perceptions of SP staff that evaltxatioa 

has T^en l^J^tinized in the distr^t. Althou^ probles;s continue in 

various areas such as providid^ tioely feedback o£ Unfbrr^tion « 2sain* 

talning cozsanication abcnit intent of evaluation and care^l laonitoring 



of process tx> detezadM icpleceatatioii, it:. Is apparent that evaluatica 
can be a credible process. Tc?e rasidate for Title I eraluatlOT iias 
served to structure not coly e3q>ecta2icies for evalnatioa of Title I 
projects but for other district endeavors as veil. Hc^efuliy, the 
evaloatioa effort has desoast rated the utility of evaluation services 
for yro^iTZJi iiiprovenent. Tsx^ jioral of the story seens to be that 
tztility vill soon be discovwed^ The vord €5^>ears to get around very 
f|st vhen such ser»'ices live xsp to their* proadse* 
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CCI staff 



APPHJDIX A ^ 



SrniTTOTy of Respoas&s 
K = 16 



During the past three sdiool years, have you requested assistai^ce 
froa the Division 6f Research and DcveJopnentT H ^ 

H " %' K ^ no. R ^* 

Yes 14 87,S Ko J 6.-3 1 6^3 

If no, please skip to question f^j. 

a. If yes"^ please indicate in which of the areas listed below 
assistance was requested. {Please check itca.) 

H % K % 

6 37,5 (1) writing l>ehavioral . 3 18>8 C^J desiping cjuestionnaires, 

objectives surveys and interview 

K % ^ , schedules 

_ 4. 25^. (2) constructing class- H % 

Tocm tests 3 it\^{7) compiling infonsation 

K % * froa school records, 

4 25. <3) selecting standard- personnel or sources 

ired tests N % ' 

K * 6 37>S(8) conducting^aff 

13 81 > 3 (4) writing project dcvelopcrent prograias 

proposals N % 

K % 61 37.5 (9)^ researching particular 

7 43JB (5) conducting a program topic or question 

evaluation H % 

3 38.8 (10) other: critiquing Baterials 

w . preparing reports 

In general, how welb satisfied were you- with tHe servicesS>f the. 
Division in the areas for which you had assistance? Place check belov» 

Very Kell Well SoDcwhat Hot ^Not at all No 
- Satisfied Satisfied Satisfied Satisfied Satisfied Response 

N% N%N% ' 
12 7S : 2 12>5 ; 1- .63 ' : I - 1 63 



Coraents: 
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CCl Staff 



5. If tJie Division has cqreiuctcd a progran evaluation for yoa, i«wld 
yoa request sucii service again? Place a dieck in tise blank below. 

Ko \ 

Ifadecided Probably Ko ttefiniteiy Ko Responsej 

H * 

' 7 43.; 



Definitely Yea Probably Yes 
K % 
9 S6.3 : 



4. Have you received copies of evaluation reports prepared by this 
Division of the following prograissT K \ 

H \ - H % Ko. R 
^ Yes 12 75. Ko 3 18. 8 1 6.3* 



If yes (Please check any in the list below vhidi yoa teve received.) 

• * 

If no, please skip to question ei^t ^ 

(For exar^Ie; Qiild Development, kSi Skills. 



H % 

8 SO.O Title I progxass 



N * 

7 43>.8 Title III prograr^ 
N % 

1 6^ Title VI prograias'' 
N % 

2 YZ.S Title VII prograsis 

% 

4 2 5,0 Title IX prograns 



Reading Iirprovcacntjand the 
lihe) 

CEnvironiaental Action, Project li:pact)and 
the like) 

(Handicapped Children programs) 
(Bilingual prograa) 
(Ethnic Studies progran) 



S. Please list any other evaluation reports froa this Division vhich yoa 
have received. 



T 



1 



6/ l]ou vould you describe your reactions to these evaluations or others 
fron the Division which you have ha3 a chance to read. Place a cheisk 

Ko ' 
Ea$i5 
for 



in the T) lank which is closest to the way you ^eel* 
N 



% N.%K%N%NV 
• " 3 

positive 9 51.3 :I8,7:. : - negative 

113 3 ^ ^ 

indifferent 6,3^/6.3:i8.7>/18.7 ifntbusiastic^^ 

supportive 9 S6.3: ; 2 12. 51 : i . opposed 

useful' 9 56.3 ;3 18v 8: t t useless'. 

1 S - 

discouraging . 1/6.3:31. 3:4' 25. 0 proaisinjg 

, Irrelevant. ' : t ;5*18.7*8 50^ 0 relevant 

4 1 r ' _ 

clear 7 43.8 : 25.0:/6.3: : j_ " confiisin^ - 

' •> >6 

effective *6 37.5 t^37;5: , -\ ineffective 

ERJC ' interesting? 43.8 tli^Jni.St " : dnlt 



aosver 



^ r 



i ' CCI Sta££ 



Do yoo fool that any of these reports cyf fcr rodcls for evaluation 
of progrEffis in your cuxri culua areat 

Yes. Ho 

Co32Dents: ' . 



Khat do ycai consider to bp the prlnary purpose of evaluation of 
school and/or subject area prograiKS? 



Khat changes would you reconnnend that this Division linplement' in 
it^ evaliiaticm operations? 



Project St&££ 



APPEKDIX A 

Stnnnaiy of Zespoos^ 
H = 29 



In y<«ir years as a project ^ianager, the Division of J^sca^ch has 
provided evaluation service to your project. In gcnejfel, htm well 
satisfied verc ^oa initially with, the services of the Bivisioa 'in 
evaluating yoax project. Place a check in the blank b slow. 



Very Well Well Scmcwhat* Little Kot 

Satisfied Satisfied Satisfied Satisfied Sati 



H 

12 



44.3 ? 9 



31.0 



N 
:3 



10-3 



K 
3 



10.3 



H 
1 



X all 
fi€d 

3.4 : 



No 
Response 



li 
X 



X 

3.4 



2. How has your opinion changed since then? 



/ 



I.bch Bore Kore No Oiange Less j-iich less 
Positive Positive Positive ftjsitive 

Now How How Jtow 

5 17.2. 4*" 1S.7.1& j52.0 . 1 3-4 . ' ; 



H 
1 



Ko- < 
Response 

% 
3*.4 



ERIC 



4, 



If your initial 'opinion has «iChanged, uhy? Yes 7 24:i% N o 22 7S.9i 
. more familiar with evaluation procedures and prc^graa ^ 

. feedback froa Division has anlved early enoa^ to you xn planning 

^ 2 . .change t>f evak»C&r who proved jaore knowledgeable, showed 
concern for project success and. cade -aore^ services availab ^to us 

. isjroved quality of reports "^bd contiga!ng service - ' 

2 . more cooperative planmng and eyaluatgi?i| Be^fte more knowl edgeablit 

about progran- and dif^culties encountered, "f.-.- . ' k ~ 

How would you describe the ^itial reactions to the cva"hiation of . - . 
, the groups listed below? (pgi^ a check fc1ie<blank >Mc.h Is closesi . ' -- 
to tb6 liay you feel.) ' : . • ' ' -. ' . - 

' - ScracKhat. . Soa^i^t Strongly 



Project staff 

. Other teachers in 
,.p^e(rt schools 

Pri^dL^ls^ izL 
project schools ' 

Parents , 

Adainistzation 

PAC/tosotalty 



1 3.4 .<3^*4S.3: « ^7,5^ 2 



4^ 


13.7 


IT— 

5 17.2 5 


.8 , 31. 6 5 S 


27v5 i 








'4 






7: 24^1 sld 




1 






10 


?4.4 






^2«4 i 




r 


: 


9 


31,0: 


5 :'X7.2 5 


9 31.<) - i 






m 


- '' • * 




•> • 


^ 5'r&2 S 


2 > 











.Project Staff - * 

5, flare tijpsc- reactixos disagcd over the past few ycaa's? Use the 
^ilowing scale by placing a ^^ck la the appropriate blank belowj 

Sasts 

Grown more ■' Grown Less for ^ 

Favorable OiaEge pavorablo answer Respoose 

K, % n \ n. i n % H i. 

^10 S4.S : 14 48.2 i ^ 6.8 ; 2 6.8: 1 3.4 " 



Project stiff 

Other teachers in project 
schools 

Principals in project schools 
Parents 

Adttinistratioa 
PAC/Comnamity Councils 



8 27. S : 14 48--2 S 




: 2 6.8: S 17.2 
T6.g? 4 13.7 
: 3 10.5:11 37.9 



; 3 10-3: 9 31 >0 
: 4 13.8:13: 44>8 



Ho 



Response 




24- 82.8 




25 86.2 




21 72.4 


8 


26 89.5 


3 


22 75. 8 


7 



6. Khich of the following factors probably cxxntributed to any resistance 
or negative re^tioa to tixe project evaluation froa the staff? Please 
dieck below. * 



>^ ^ ■ , 

2 . inherent suspicion of evaluation 

.1 Other (specify) , poor testing situation set up ty evaluato r 

. lack 5f understanding about necessity for 

prograa evaluation 

. lateness of availability of results 
22 75*8 T ?£ii resistance was encountered * * * 

. teachers find it haiti to distinguish* between K 
Don't taiov . project evaluation and individual evaluation' 

\^ \ . staff vas on defensive because of havii^g-feeen 

scolded about school standings 

7. Please indicate by a che^Jc if you hstve used any of tiiB evaluation findiaga 
in .discussing the prt^ect In • (Process. Evaltiation) 

28 96.5 ^af f conf erences>, meetings 

^ " 13 ^t^AC Bcetiiigs . • ' 

9 31.0 PTA Boetingp ^ ^ 

11 37.9 Parental visitations to the project classes, seijficc^ 

. ^fhrinistratlve personnel , , coacunity 
♦ IS SS.l Other iSSSCIFf) . State*aeetin*gs» SEA co^rrtmications council 



u 



deaonstrati<^ aeetings . Federal officials 

. salesaiea trying to sell vaies- .'agencies . 



ERIC 



d 5.4 Hot os^ 



. insefvice ceetings^- iirbrkshqps 
. hooe visitatidns 
• national xieetings 



27 



■ Project Staff 

4, Ikw veil is jc^gch 6£ tf» follcviag aspects of yoar project 'fecscribfcd 
ia.'tSK cvalnatioQS of yoar project? Place a dieck ia tiie hlsnii -iMch 
Is closest to the way jwr feel* . Ko 

Basis 

V«y Fairly Hc« at for So 

, ^ Well Beil all saswex a&g)oase 

I . * ^ S * 8 . i -13:.» .J< V. S. % H-T — H — 5 

affective effects 4'i5.7i 6 20.6 ! / •44.a 1 3,4^; ; 1 3.4 ; 4 13.7 

cognitive effects K 41.3 ? 7 24. g 6 20. g : ; 2 6.8 ; 2 6.8 

staff opinion ~ 1 1 57.9 : 6 23.6 ; 6 20.6 ? 1 3.4 : 3 IO.5 : ; 2 6.8 

pareatal opinion 7 24.i;i 3.4 ; 7 24.l t 2 6.8 : 5 17.2: 2 6.8 : S 17.2 

cosmmity c^inioa 4 13.7 t : 4 1317; 4 13.7; 6 2Q.6 ; S 17.2: 6 20.6 

cost effectiveness 6 20.6 ; T 17. g 3 10.3:2 6.8:3 10.3:3 10.3: 7 24.1 



staff -developjiient 

activities 9 31.0 : / .5*Tlfr3 1t).^3 10.3:2 ' 6.8: 



9 31.0 : / ^£%^W^ : 
13 44.8": 7 24* l: 3 10.3:; 



instructicgial procedures 13 44.8 : 7 24.1: 3 10.3:_2 6.S;. :1 3.4 



■project timetable' 8 Z7.5 ; 5 17.:^l"l0.3t 1 3.4:2 6.8:4 13.7 



disseninaticn activities 8 27.S : 6 20.6;3 10.3; 2 6.8:4 13.7:2 6.8 



popil selection 14 48.3 : S 17.2:1 3.4:3 10.3:1 3.4:3 10.3 



evaluation plan 1 8 62 ..0 : 5 17.2:3 10.5 1 g.4 : 



data analysis r6 55.1 : 7 24. g 2" 6.8r. : ■ : 1 3-4 



stafT respoasibiliti'es 1 3 44.8: 6 20.'g 5 17.2:1 5.4M 3.4: 
project materials 9 31.0 : 9 31.0:5 17.2^y 3.4 ^ 2 6.8? 



6.8 



3 10.3 



6 20.6- 



4 13.7 



6.8 • 



6.8 



3 10.-3 



3 10.3 



3 10.3 



project organization ^ 

and oanagenent . 1 5 51.7: 4 13.7:5 17.2:2 6.^1 3.4: : 2 6.8 

• • • " 

instructional techniques 7, 24.1 : 6 20.6:6 20.6:1 3.4;S 17.2rl 5>4 ; 3 10.3 

SI How would you describe your overall reaction to tie evaluations 

provided for yotn^ pryject? Place a check, in t2xe blank vhich is closest 
to the. way you feel. , ' ' . Ko'gespoiase 

' ' • A> . . 

N %i N % H % H % N % . - * . 

indifferent 1^4^ . S 17,2:9 51.0;10 5 4.4 entlif>siasti« 4 13:7 * " • 

V- ■ - 

- useful I S 51.^:9 31.0;2 6.8;1 3*4;- 1 5.4 us^css 1 3.4 

> ^^^^^^^^^ 

effective 12 f 41.3;^ 27.5:2 6.8;1 3>4; 2 .ineffective 4 13,7 

. discomaging 1 3. 4 il 5:4:6 20.6:6 20.6: 12 4 1.3 pronising 510,5 /''^sm^ 

izxelcys^ . ^ 6.8; S 11,2:6 20.6:13 44.S rclovant- 3 10.3 , • 
AJ^ : «— —r->-^ - 1 

clear 1 3 4#8£7 17.2;4 13, ; 7;4 13.7; confcsin^ 1 3.4 . ^ . 

intcrestint lO 34.4rl0 34.0:4 15.>7; 3 lO.s ^uU * - 2 6.8 \ ':2 

lj*glt£V* . i 3.4^ I 3.y;4 l3.7Vi 27.S?13 4 4:.8 pcJflitlv* 2 §.4- 



Project Staff 



Bare yoa fcxrjd t3at the craluations sapply ixa.viti infornatloo'^F' 
decision jsaiicg aboat yoatr project? 

_ * • Y« Ko 



^12i2t do yoa cpasidcr to be the primary p:iTpose of eraluatipn of 
projoctsT 



ifhat changes voaid you reconstcnd that this Dirisioa iitpleroent in 
its evaluation operations? 

\ 
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